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How NAEP Frameworks Serve the Public Good 



National Assessment for Educational Progress (NAEP) frameworks describe the content 
and skills measured on NAEP assessments as well as the design of the assessment. They provide 
both the “what” and the “how” for national assessment. Representing the best thinking of 
thousands of educators, experts, parents, and policymakers, NAEP frameworks describe a broad 
range of what students learn and the skills they can demonstrate in reading, mathematics, 
writing, science, history, civics, economics, foreign language, geography, and the arts. NAEP is 
the only nationally representative and continuing assessment of what America’s students know 
and can do in various subject areas. The frameworks represent a vision for America’s future. 

The hopes expressed in the Civics Assessment Framework are at the heart of every 
NAEP initiative. “The assessment is designed to show how well American students are being 
prepared for citizenship in our constitutional democracy . . . the National Assessment Governing 
Board hopes its results will be used to improve civic education for all of America’s children and 
to help make sure that our republic, established near the end of the 18th century, continues alive 
and well into the 21st and beyond.” 

NAEP frameworks serve the public good in many ways. The process used for their 
development creates a democratic dynamic forum for experts to address issues in their field. 
Varying opinions are sought and aired. Working together, thinking together, and learning 
together, framework committees construct a document that represents shared beliefs and a shared 
description of what students should know and be able to do. Although discussion during the 
process may sometimes become contentious, in the history of the Governing Board every 
framework has been adopted by the Board unanimously with no dissenting reports. 

NAEP frameworks also help to inform public discourse. By making transparent the 
content and skills measured on NAEP assessments, they allow the public to see what is behind 
the scores. This is of critical importance when The Nation’s Report Card is released and 
comparisons of student achievement are drawn across states. NAEP frameworks help leaders 
across the nation make good decisions about state and local standards and assessments. In order 
to measure trends in student performance, NAEP frameworks are designed to remain stable for at 
least 10 years. At the same time, all frameworks are responsive to changes in national and 
international standards as well as in curriculum. Without advocating any particular approach to 
instruction or attempting to represent the full range of what should be taught, NAEP assessment 
frameworks provide a starting place for constructive conversations about improving education. 

What Is a NAEP Assessment Framework? 

NAEP frameworks describe the assessment objectives and design for national tests in 
reading, mathematics, writing, science, history, civics, economics, foreign languages, geography, 
and the arts. Governing Board policy dictates that these assessments must be valid, reliable, and 
based on widely accepted professional standards. Since 1989, the Governing Board has 
developed 15 frameworks in 10 subject areas. 
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Another way to define a NAEP framework is to consider what it is not. Designed to 
shape the content and format of NAEP assessment across the content areas, NAEP frameworks 
are not curriculum frameworks. Unlike national and state standards that provide a comprehensive 
picture of what should be taught, NAEP frameworks describe what should be assessed at grades 
4, 8, and 12. They must also take into account the constraints of large-scale assessment. Many 
important components of a school curriculum (for example, research projects and collaborative 
presentations) do not appear in NAEP frameworks because within the current limitations of time 
and resources, such aspects of a curriculum cannot be assessed on NAEP assessments in a valid 
and reliable manner. This should not suggest that such demonstrations of student learning are not 
important. As new technologies enlarge our ability to measure student learning, other aspects 
may open up for future NAEP assessment. 

Who Is the Audience for NAEP Frameworks? 

The primary audience for a NAEP framework is the general public. A challenge every 
NAEP framework development committee faces is finding ways to describe the subject area in 
specific detail. Committee members try to employ the vocabulary of the discipline without using 
language that excludes those who do not work in the field. A good example of how this 
challenge has been met can be found in the 2009 Mathematics Framework. Notice how carefully 
writers of the framework define mathematical terms within their explanation of what students 
should know and be able to do with data analysis and statistics: 

Data analysis and statistics cover the entire process of collecting, organizing, 
summarizing, and interpreting data. This is the heart of the discipline called statistics, 
and is in evidence whenever quantitative information is used to determine a course of 
action. To emphasize the spirit of statistical thinking, data analysis should begin with 
a question to be answered — not with the data. Data should be collected only with a 
specific question or questions in mind and only after a plan (usually called a design) 
for collecting data relevant to the question is thought out. Beginning at an early age, 
students should grasp the fundamental principle that looking for questions in an 
existing data set is far different from the scientific method of collecting data to verily 
or refute a well-posed question. A pattern can be found in almost any data set if one 
looks hard enough, but a pattern discovered in this way is often meaningless, 
especially from the point of view of statistical inference (p. 29). 

Writers of frameworks work to ensure that the language they use to describe educational 
objectives communicates clearly to local school board members and parents. To serve the public 
good, frameworks must be written in language that illuminates, not obfuscates, the issues at 
hand. 



Test developers constitute a second audience for NAEP frameworks. For these users of 
the assessment framework, the planning committee creates a blueprint for the assessment. 
Planning committees invest enormous time and effort explaining and offering examples of the 
kinds of questions they believe will best assess the content and skills described in the framework. 
Test specifications recommend how content and skills should be grouped and the distribution of 
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items within each content area category. For example, the 2009 Science Framework for grade 8 
states that the distribution of items should be: 



• 30 percent for physical science 

• 30 percent for life science 

• 40 percent for Earth and space sciences 

The emphasis on Earth and space sciences for grade 8 is an increase from 33.3 percent in 
grade 4; it decreases to 25 percent for grade 12. Such decisions about grouping and weighting of 
test items reflect hours of deliberation and careful consideration of differing points of view. 

To ensure that test developers understand exactly what steering and planning committee 
members intend, frameworks include illustrative items. The 2009 Science Framework cited an 
example from the Trends in International Mathematics and Science Study grade 4 assessment to 
demonstrate the kind of short constructed response item that NAEP might include: 

Illustrative Item 



On a hot, humid day the air contains a lot of water vapor. What happens to the 
water vapor in the air when the air becomes very cold? 

A scoring guide for the item appears in the Science Framework appendix. Along with 
providing guidance to test developers, examples and scoring guides offer the general public and 
other users and consumers of NAEP a window into the test development process. 

During deliberations for development of the 2009 Reading Framework, vocabulary was 
identified as a critical component of reading comprehension that should be reported separately. 
After intense debate about how vocabulary might be measured on NAEP assessments, committee 
members crafted the following explanation to guide test developers: 

Vocabulary items will be developed about the meaning of words as they are used in the 
context of the passages that students read. Students will not be asked to draw on their 
prior knowledge by providing a written definition of each word on a list or in a set of 
words. There are two reasons for this approach. First, knowledge as explicit as a written 
definition of a word is not the specific ability required for passage comprehension. In 
reality, readers may not be able to provide a complete definition of a word they 
encounter but do have enough of the sense of the word’s meaning as used in text, that 
their comprehension is not impeded. A second argument against requiring specific 
definitions is that word meaning often depends on the context in which the word 
appears (p. 34). 

You will notice that the 2009 Reading Framework provides a rationale for the 
committee’s decision about how vocabulary should be assessed. Such transparency contributes to 
why NAEP frameworks and assessments are considered a “gold standard.” 



3 




How Is a NAEP Assessment Framework Developed? 



Before designing an assessment, one needs to decide what is to be measured. NAEP 
assessment frameworks identify the content and skills that can be assessed on a large-scale 
national test. The framework also provides guidelines for the design and fonnat of the 
assessment in the form of test specifications. It is an enormous task to determine exactly what 
content and skills should be included. 

The framework development process takes from 1 8 to 24 months and involves a host of 
individuals drawn from professional organizations, schools, universities, the business 
community, and the military. Participants in the framework development process include 
nationally known researchers on the cutting edge of cognitive research as well as experts in 
assessment, teachers, state and school district representatives, policymakers, and members of the 
general public. Over the years participants have included a former Governor, a news anchor, 
reporters, business chief executive officers, practicing artists, a law enforcement official, a 
children’s book author, historians, engineers, scientists, financial managers, mathematicians, 
human resource representatives, an Air Force instructor, and a member of the National Security 
Agency. Though diverse in background and experience, participants share a common goal — to 
serve America’s children. 

A steering committee and a planning committee work in tandem. The steering group 
offers direction while the planning group drafts the framework. At every stage NAGB’s 
Assessment Development Committee and the Board provide input and guidance. Drafts of the 
document are sent out for review to a wide range of stakeholders, all of them users and 
consumers of NAEP frameworks and data. The framework is ultimately sent to the Governing 
Board for adoption. 

At every stage of the process, NAEP framework development is comprehensive, 
inclusive, and deliberative. The first stage is a competitive bidding process to detennine a 
contractor for framework development. Contractors often collaborate with professional 
organizations such as the National Council on Economic Education, the Association of State 
Supervisors of Mathematics, or the National Science Teachers Association. Over the history of 
framework development, contracts have been awarded to the American Institutes for Research; 
American College Testing; the College Board; the Council of Chief State School Officers; the 
National Center for Research on Evaluation, Standards, and Student Testing (CRESST) at 
UCLA; and WestEd, among others. 

The Governing Board then gathers recommendations for framework committee members. 
When NAEP first began developing frameworks it was sometimes difficult to persuade leaders 
in the field to participate — members are not paid for their contribution, only reimbursed for 
travel. NAEP’s growing influence as well as growing respect for the integrity of the framework 
development process has resulted in a 98 percent acceptance rate for recent invitations. 

Before the first meeting of the steering and planning committees, the Governing Board 
commissions an issues paper. Written by one or more prominent researchers in the subject area, 
the paper serves as a basis for the committees’ discussion. Issues papers raise important 
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questions that help to frame steering and planning committee deliberations on critical concerns in 
the field. The paper also provides framework committee members with recent research germane 
to their task. For the 2009 Reading Framework, the Board commissioned two separate issues 
papers, one from the American Institutes for Research and another from the Educational Leaders 
Council and the Thomas B. Fordham Institute. The goal was to ensure that a broad range of 
perspectives informed framework deliberations. For the 2009 Science Framework, the Board 
convened a panel whose charge was “to identify issues emerging from the nation’s requirements 
for a science literate citizenry and perspectives on science literacy in documents developed by 
the U.S. and international science education communities” (p. 1). The panel determined that the 
essential challenge facing framework committees is the breadth of science content, 
understandings, and abilities that might be included in the Science Framework. For the 2009 
Writing Framework, the issues paper posed eight questions for committee members to consider: 

1 . What types of writing should be assessed, and how are they related to one another? 

2. What writing tasks/types should be assessed at each grade level? 

3. How can the 12th grade assessment be structured to measure preparedness for 
postsecondary endeavors, including college, workplace training, and entrance into 
the military? 

4. Should the writing assessment be computerized? 

5. What aspects of writing achievement should be measured? 

6. What should students write about? 

7. How should the framework address the question of time? 

8. What accommodations should be made for English language learners, students with 
disabilities, and low-achieving students? 

The Writing Framework issues paper concluded, “Most of the issues outlined above 
have no easy answers, but the collective experience of the members of the Framework 
Committees offers an opportunity to make significant progress in addressing them” (p. 14). 

This sense of opportunity is the spirit that animates both the creation and the application of 
issues papers. At every stage, framework development is informed by research in the field 
and in education, best practices, international assessment frameworks, and state standards. 

What follows for steering and planning committee members is a year of meetings, 
conference calls, and e-mail communication. Guided by the framework contractor and the 
Board, small groups of committee members are asked to focus on particular aspects of the 
larger task. For example, in the 2009 Reading Framework deliberations, one group worked 
to determine the kinds of texts students should be asked to read for the assessment. For the 
2011 Writing Framework, a small group focused on the implications of a computer-based 
writing assessment. Following group deliberations, committee members regroup to report on 
the progress of their thinking. Technical experts sometimes attend small group sessions to 
help inform the discussion in terms of research or aspects of NAEP assessment with which 
committee members may be unfamiliar. 

Once a working draft of the assessment framework is in place, the contractor and the 
Board members hold public forums to obtain feedback from users and consumers. They reach 
out to the business community, testing directors, parents, and students to elicit comments and 
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concerns. The contractor and the Governing Board report these responses and concerns to the 
committee members, who take them into account during their revision of the framework. Drafts 
are also sent out for review to professional organizations, school districts, and business groups. 
Over 60 organizations were invited to comment on the initial 2011 Writing Framework 
recommendations. 

Before the framework reaches its final form, the Board holds public hearings. These 
sessions allow Board members to hear testimony for and against the proposed framework. The 
proposed framework is posted on the NAEP Web site in order to gamer additional commentary 
from interested parties. The Board then summarizes and presents this commentary to the Board 
and to framework committees for discussion, reflection, and revision. Frequently, the framework 
committee generates and submits several dozen drafts. After many reworkings, the framework is 
sent out to an external review panel, which is typically a small, prestigious group of leading 
researchers in the field who have not been involved in the development of the framework. They 
are asked to review the draft and comment on its strengths and weaknesses. The purpose of this 
extensive outreach is to ensure that as many perspectives as possible are considered before the 
Board votes on a new framework for adoption. 

How Does Governing Board Policy Determine the Development of NAEP Frameworks? 

The Governing Board framework development policy states, “Since its creation by 
Congress in 1988, the National Assessment Governing Board has been responsible for 
determining the content and format of all NAEP subject area assessments. The Board has carried 
out this important statutory responsibility by engaging a broad spectrum of educators, 
policymakers, business representatives, and members of the general public in developing 
recommendations for the knowledge and skills NAEP should assess in various grades and 
subject areas. From this comprehensive process, the Board develops an assessment framework to 
outline the content and fonnat for each NAEP subject area assessment” (pp. 1-2). 

The following principles guide framework development: 

• Principle 1 — The Governing Board is responsible for developing an assessment 
framework for each NAEP subject area. The framework shall define the scope of 
the domain to be measured by delineating the knowledge and skills to be assessed 
at each grade, the fonnat of the NAEP assessment, and preliminary achievement 
level descriptions. 

• Principle 2 — The Governing Board shall develop an assessment framework 
through a comprehensive, inclusive, and deliberative process that involves the 
active participation of teachers, curriculum specialists, local school 
administrators, parents, and members of the public. 

• Principle 3 — The framework development process shall take into account state 
and local curricula and assessments, widely accepted professional standards, 
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exemplary research, international standards and assessments, and other pertinent 
factors and information. 

• Principle 4 — The Governing Board, through its assessment development 
committee, shall closely monitor all steps in the framework development process. 
The result of this process shall be recommendations for Board action in the form 
of three key documents: the assessment framework, assessment and item 
specifications, and background variables that relate to the subject being assessed. 

• Principle 5 — Through the framework development process, preliminary 
achievement level descriptions shall be created for each grade being assessed. 
These preliminary descriptions shall be an important consideration in the item 
development process and will be used to begin the achievement level setting 
process. 

• Principle 6 — The specifications document shall be developed during the 
framework process for use by NCES and the test development contractor as the 
blueprint for constructing the NAEP assessment and items in a given subject area. 

• Principle 7 — Name assessment frameworks and test specifications generally shall 
remain stable for at least 10 years. 

Since 1969, NAEP has reported nationwide data, based on nationally-representative 
samples, on what students know and can do in elementary and secondary schools. More recently, 
NAEP has reported student results by state as well as for a set of large, urban school districts. By 
law, NAEP must provide, in a timely manner, a fair and accurate measurement of student 
academic achievement and report trends in achievement in reading, mathematics, and other 
subjects (Public Law 107-279). NAEP assessment frameworks ensure that student achievement 
is measured fairly and accurately. 

In addition to creating the assessment framework and test specifications blueprint, 
committee members make recommendations to the contractor for background variables 
collected in conjunction with the particular content area. As stated in the Board’s policy, 
background data on students, teachers, and schools must fulfill the statutory requirement 
that NAEP reports include information, whenever feasible, disaggregated by race or 
ethnicity, socioeconomic status, gender, disability, and limited English proficiency. The 
information provided by background variable data enriches the reporting and understanding 
of NAEP results. 

Assessment frameworks must also include preliminary achievement level descriptions for 
each grade at the Basic, Proficient, and Advanced levels. Achievement levels describe the degree 
to which student perfonnance meets the standards set for what students should know and be able 
to do. Framework committees take the Governing Board’s policy definitions for Basic, 
Proficient, and Advanced achievement and develop content-specific descriptions of student 
achievement for grades 4, 8, and 12: 
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Basic 

This level denotes partial mastery of prerequisite knowledge and skills that are 
fundamental for proficient work at each grade level. 

Proficient 

This level represents solid academic perfonnance for each grade assessed. 

Students reaching this level have demonstrated competency over challenging 
subject matter, including subject-matter knowledge, application of such 
knowledge to real-world situations, and analytical skills appropriate to the subject 
matter. 

Advanced 

This level signifies superior perfonnance. 

To see how framework committees use the Board’s policy to shape content-specific 
preliminary achievement level descriptions, examine the 2009 Mathematics Framework 
description of Proficient achievement in grade 8: 

Eighth-grade students performing at the Proficient level should apply 
mathematical concepts and procedures consistently to complex problems in 
the five NAEP content areas. 

Eighth graders perfonning at the Proficient level should be able to conjecture, 
defend their ideas, and give supporting examples. They should understand the 
connections among fractions, percents, decimals, and other mathematical topics 
such as algebra and functions. Students at this level are expected to have a 
thorough understanding of Basic level arithmetic operations — an understanding 
sufficient for problem solving in practical situations. 

Quantity and spatial relationships in problem solving and reasoning should be 
familiar to them, and they should be able to convey underlying reasoning skills 
beyond the level of arithmetic. They should be able to compare and contrast 
mathematical ideas and generate their own examples. These students should make 
inferences from data and graphs; apply properties of informal geometry; and 
accurately use the tools of technology. Students at this level should understand the 
process of gathering and organizing data and be able to calculate, evaluate, and 
communicate results within the domain of statistics and probability (p. 78). 

Achievement levels are cumulative; that is, students perfonning at the Proficient level 
also possess the competencies associated with the Basic level, and students performing at the 
Advanced level demonstrate the skills and knowledge associated with both the Basic and 
Proficient levels. 
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What Content-Specific Issues Must NAEP Framework Committees Address? 



Each subject area has its own set of dynamics and its own controversial issues. Given the 
different backgrounds that steering and planning committee members bring to the table, it would 
be naive to expect experts to agree on every issue. Fortunately, the framework development 
process has a built-in system of checks and balances that allows for a free and open exchange of 
views as well as a means for finding common ground. Following are descriptions of some of the 
issues framework committees have addressed in the past. They illustrate the complexities 
involved in the task of trying to describe what students should know and be able to do. 

2009 Mathematics Framework 

When the steering and planning committees for the 2009 Mathematics Framework 
convened, the Board had for several years been considering how to improve 12th grade 
achievement. In 2003, the Governing Board charged the National Commission on NAEP 12th 
Grade Assessment and Reporting to review current 12th grade assessment and make 
recommendations for improvement. The commission’s report, 12th Grade Student Achievement 
in America: A New Vision for NAEP, recommended that 12th grade NAEP be redesigned to 
“report on the readiness of 12th graders for college, training for employment, and entrance into 
the military” (p. 1). 

To implement this recommendation in the 2009 Mathematics Framework, the Governing 
Board contracted with Achieve, Inc., to examine the 2005 Mathematics Assessment Framework 
in relation to benchmarks set by the American Diploma Project. A panel of mathematicians, 
classroom teachers, and policymakers proposed increasing the scope and rigor of the grade 12 
NAEP assessment. Achieve developed new assessment objectives; a panel of mathematicians 
and mathematics educators reviewed and revised the objectives, matching them against current 
objectives for grades 4 and 8. The panel convened focus groups with the Association of State 
Supervisors of Mathematics to obtain feedback on the proposed objectives and sent out surveys 
to various NAEP stakeholders. Both processes involved repeated rounds of reviews. In August 
2006, the Governing Board approved the final set of new grade 12 objectives for mathematics. 

Mathematics framework committees, along with other framework committees, address 
the disparity between what is and what should be. For the grade 12 framework, this disparity 
required certain objectives to be marked with an asterisk, indicating that they describe 
mathematics content beyond what is commonly taught in a standard 3-year course of study (2 
years of algebra and 1 year of geometry). The asterisk designates that for the present these 
objectives will appear on NAEP assessments less frequently than others. An example of such a 
designated objective from data analysis, statistics, and probability is: “Recognize the differences 
in design and in conclusions between randomized experiments and observational studies.” An 
example from the algebra objectives is: “Analyze properties of exponential, logarithmic, and 
rational functions.” These designated objectives describe what mathematics experts and 
mathematics educators have detennined that grade 12 students need to know and be able to do in 
order to be prepared for the mathematical demands of the 21st century. Including them in the 
assessment framework will help us move from “what is” to “what should be.” 
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2009 Reading Framework 



As demonstrated by the commission of two separate issues papers, the 2009 Reading 
Framework committees needed to resolve widely divergent views on how NAEP should assess 
reading. In order to help committee members frame the issues clearly, the Board prepared a 
comparison of the points made in the issues papers, beginning with their definitions of reading. 
To the enormous credit of everyone involved in the construction of the Reading Framework, the 
final version of the document represents a meeting of minds as well as a coherent description of 
assessment objectives for the 2009 NAEP Reading Assessment. 

Along with the issue of vocabulary assessment described earlier, framework committee 
members had many decisions to make about the kinds of texts to be included on the assessment. 
Research on reading suggests that literary and information texts pose very different challenges 
for readers. Both kinds of text, therefore, will be measured on the assessment. Although some 
committee members felt strongly that computer-based electronic text should be included on the 
assessment, the group agreed that as long as the NAEP Reading Assessment employed a paper- 
and-pencil format, it would not be possible to present electronic text to students authentically. As 
a computer-based reading assessment becomes more feasible, this issue is likely to be 
reconsidered. 

Another aspect of text selection for reading passages on the assessment involved the 
weighting of literary and informational texts across the grade levels. While literary texts continue 
to dominate courses of study in English classes, the assessment is designed to measure reading 
across the content areas. Responding to the Board’s report recommending that grade 12 NAEP 
assess postsecondary and workplace preparedness, informational text in the 2009 Reading 
Framework is increasingly emphasized for grades 4, 8, and 12. 

2009 Science Framework 

New national standards for science literacy, advances in science, and research into 
cognition, together with the development of innovative assessment approaches, made the 
revision of the 1991 NAEP Science Framework an idea whose time had come. In an effort to 
keep the nation’s youth competitive internationally in science and technology, committee 
members scrutinized international assessment frameworks, such as the Trends in International 
Mathematics and Science Study (TIMSS) and the Program for International Student Assessment 
(PISA). The challenge they faced was how to create a framework for the future in science. The 
framework was designed to strike a balance between what can reasonably be predicted about 
future school science and what students are likely to encounter in their curriculum and 
instruction now and in the near future. The framework is intended to be both forward looking 
and reflective: forward looking in terms of what science content will be of central importance in 
the future and reflective in terms of current school science. 

The 2009 Science Framework describes new ways to measure scientific literacy. New 
assessment items will involve the use of interactive computer tasks. Recommended items focus 
on students’ conceptual understanding — their knowledge and use of science facts, concepts, 
principles, laws, and theories — and measure students’ ability to engage in the components of 
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scientific inquiry and technological design. The framework also describes new item types, 
including computer simulations for a subset of students who will receive an additional 30 
minutes to complete hands-on performance on interactive computer tasks. Interactive computer 
tasks may be of four types: 

• Information search and analysis items pose a scientific problem and ask students to 
use an information database and analyze relevant data to address it. 

• Empirical investigation items put hands-on performance tasks on the computer and 
invite students to design and conduct a study in order to draw conclusions. 

• Simulation items model systems such as food webs and ask students to manipulate 
variables to predict and explain the resulting changes. 

• Concept map items probe aspects of the structure or organization of students’ 
scientific knowledge by providing concept terms and asking students to create a 
logical graphic organizer. 

The 2009 Science Framework also includes questions on technological design, which is 
the systematic process of applying science knowledge and skills to solve real-world problems. It 
describes “the kinds of tasks, problems, and exercises, along with the kinds of knowledge and 
reasoning, that should be expected of students as a result of what is taught in the science 
curriculum, consistent with the National Standards and Benchmarks” (p. 92). The framework 
represents the kind of scientific literacy that students need so they can be ready for 
postsecondary education and the workplace. 

2011 Writing Framework 

From their first meeting, members of the 2011 Writing Framework committees wrestled 
with the issue of how communication technologies affect the ways in which people write. The 
21st century places new demands on us, specifically the need to write in a timely and efficient 
manner. For example, composition frequently takes place at a keyboard. One committee member 
coined the phrase “the velocity of writing” to describe the pace of written communication in 
today’s world. A constant undercurrent in deliberations was, “How can students meet the 
challenges of 21st century writing without loss of a sense of audience, purpose, and 
complexity?” 

In response to developments surrounding the production of writing and after intense 
discussion of the implications of their decision, the framework committees recommended that 
writing at grades 8 and 12 be assessed using word -processing software with commonly available 
tools enabled. Preventing students from using spell-check tools on a computer would be like 
removing the erasers from their pencils. A great deal of thinking and research informed the 
decision not to assess grade 4 students’ writing on computers on the 2011 assessment. Although 
grade 4 students may currently lack the keyboarding expertise and experience necessary for 
assessing their writing on a computer, the 2011 Writing Framework looks forward to a 
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computer-based assessment of writing at grade 4 by 2019. The committees also recommended a 
special study to investigate fourth graders’ keyboarding proficiency as well as their experience 
with composing on computers. The dramatic shift in the tool used to assess writing from paper 
and pencil to computer keyboards “reflects the way today’s students compose — and are expected 
to compose — particularly as they move into postsecondary settings” (p. 2). 

Rather than identifying distinct types of writing as the previous framework had done, the 
2011 Writing Framework describes three communicative purposes for writing: to persuade, to 
explain, and to convey experience. For each task on the assessment, the purpose should be 
clearly stated and consistent with the intended audience for the piece of writing. Writing 
framework committees also addressed the issue of which features of writing should be evaluated. 
The three broad features according to which student writing will be evaluated are: development 
of ideas, organization of ideas, and language facility and conventions. These features are 
consistent with state curriculum standards and state assessments of direct writing in grades 4, 8, 
and 12. They are also consistent with writing expectations for college, the military, and the 
workplace. 

1997 Arts Education Assessment Framework 

Members of the Arts Education Framework committee struggled, as many other 
framework committees do, to find a balance between “what is” and “what should be.” The 
published framework attempted to combine realism with vision. While acknowledging that arts 
are often a marginal experience for students at the elementary and middle school levels, and are 
elective in high school, committee members maintained that every student should participate in 
the arts. “No child in an American school should be deprived of the opportunity to see, hear, 
touch, and understand the accumulated wisdom of our artistic heritage, and to make his or her 
own contributions” (p. 2). 

The gaping disparity among students’ experience made the task of constructing a NAEP 
assessment in arts education particularly complex. How can student achievement be measured in 
a timely, cost-efficient manner when students’ opportunities to learn have been so very different? 
Central to the Arts Framework is the assertion that NAEP assessment should reinforce the 
promise of arts education for all, including those students whose physical and mental abilities 
need additional support for artistic expression. 

2006 Civics Framework 

Concurrent with civic knowledge and civic skills, the 2006 Civics Framework 
committees determined that civic disposition is one of the three critical and connected 
components of civic education. Civic disposition refers to the inclinations that pervade all 
aspects of citizenship, including “the dispositions to become an independent member of society; 
respect individual worth and human dignity; assume the personal, political, and economic 
responsibilities of a citizen; participate in civic affairs in an informed, thoughtful, and effective 
manner; and promote the healthy functioning of American constitutional democracy” (pp. x-xi). 
Civics framework committees discussed how a national assessment might measure what Alexis 
de Tocqueville called “habits of the heart.” 
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It is not possible to measure direct participatory skills such as participating in school 
elections or attending public debates; therefore, direct measurement of civic dispositions is 
beyond the scope of the NAEP assessment. To approach these skills, the Civics Framework 
recommended that assessment questions focus on students’ knowledge and explanation of the 
importance of participation and civic dispositions. For example, students may be asked to 
describe the importance of listening respectfully to the opinions of others or to explain the extent 
to which political leaders and government agencies adhere to constitutional principles. The 
assessment will never ask students questions about personal values or convictions. Framework 
committee members recommended that assessment tasks at grades 4, 8, and 12 include a broad 
range of stimulus materials for students to examine and comment upon, including excerpts from 
speeches and important civics documents, political cartoons and photographs, newspaper 
articles, and sample ballots. Extended-response questions might ask students to take a position 
on an issue after examining multiple texts. 

The Civics Framework makes clear that the need for sustained and systematic attention to 
civics education is urgent. To make their case, committee members quote Judge Learned Hand, 
“I often wonder whether we do not rest our hopes too much upon constitutions, upon laws and 
courts. These are false hopes; believe me, these are false hopes. Liberty lies in the hearts of men; 
when it dies there, no constitution, no law, no court can save it.” 

2012 Technological Literacy Framework 

In September 2008, the Governing Board contracted with WestEd to develop a NAEP 
Technological Literacy Framework. Development of this new framework will be challenging 
because there are no nationwide standards or common definition for technological literacy. 
Growing concern about American students’ ability to compete in a global marketplace and keep 
up with evolving technology lends urgency to the work. Many groups will assist WestEd in this 
endeavor — the Council of Chief State School Officers, the International Technology Education 
Association, the International Society for Technology in Education, the Partnership for 21st 
Century Skills, and the State Education Technology Directors Association, along with 
technology experts, engineers, teachers, scientists, business representatives, state and local 
policymakers, and employers from across the country, will weigh in on framework development, 
advising on the content and design of the assessment and making recommendations to the Board 
for a 2012 NAEP Technological Literacy Assessment. 

Alan J. Friedman, a physicist and member of the Governing Board, explains, “We all 
know that engineering and technologies in all fonns — including computers, communications, 
energy usage, agriculture, medicine, and transportation — affect everything we hear, see, touch, 
and eat. With this new framework and the tests it will guide, we’ll discover how well students 
today are learning to understand and use these immensely powerful tools.” The Board is 
scheduled to review the proposed Technological Literacy Framework in late 2009. 



13 




What Can States and Local Districts Learn From NAEP Assessment Frameworks and the 

Framework Assessment Process? 



States and districts intent upon improving education may extrapolate from the processes 
used for the development of NAEP frameworks and assessments. While this means that the 
journey will be long and labor intensive, anything less will ultimately be deficient. In a 
participatory democracy, diverse points of view must be sought and heard. Stakeholders must be 
invited to take part in deliberations. Experts may have at hand an enormous body of research, but 
this information must be shared in a manner that the general public can understand. Detennining 
what is best for America’s children must be a collaborative effort characterized by transparency 
and a willingness to consider the perspectives of others. As de Tocqueville explained, each new 
generation is a new people that must acquire the knowledge, leam the skills, and develop the 
dispositions to maintain and improve our constitutional democracy. We must work together to 
ensure that every student has every opportunity to acquire the knowledge and skills needed to 
survive and thrive in the 21st century. 
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